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IESSXS  LEARNED  FIELD- TESTING  AN  EXPERT  SYSTEM 
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Hanscom  Air  Force  Base 
Massachusetts,  01731-5000 

ABSTRACT 

the  past  two  years,  an  expert  system  for  forecasting  the 
wrcut  f  mrr  cf  fog  has  been  installed  at  Seymour-Johnson  AFB,  North 
Garciirw*.  Aside  from  validation  of  the  rule  base,  this  field  test 
tot  p: ov i dmi  valuable  insights  into  some  of  the  problems  that  must 
before  such  systems  are  deployed  operationally  world- 
run  is  the  necessity  for  a  satisfactory  trade-off  between 
im-c-lvetacnt  and  real-time  data  input.  In  addition,  varying 
of  tophi stication  on  the  part  of  the  users  must  be  taken 
into  •crount .  Systems  which,  like  those  used  in  meteorology,  are 
rally  dependent,  should  be  written  in  a  form  that  makes 
thw*  readily  adaptable  from  one  location  to  another.  Adaptability 
tnclvcVt  the  ability  to  use  alternate  data  sources  for  those  occa- 
tinn*  or  locations  where  standard  observations  and  measurements  are 
•v'ailable.  This  introduces  the  concept  of  maintainability, 
an  existing  system  can  be  updated  to  include  new  data 
mcKitcvz  an}  changes  in  the  knowledge  base.  Each  of  these  concepts 
tt  di*-\i*.ced  in  turn,  with  particular  examples  taken  from  the  fog 
ft*Mr<afct  ft yi tern. 


1.  INTRODUCTION 

If-  ttereeher  1985,  the  Air  Force  Geophysics  Laboratory  signed  a 
cunttact  with  GEOMET  Technologies,  Inc.  for  the  develop- 
#  '  t,i*  novl edge-based  expert  system  designed  to  assist  weather 
.**'*"*  ln  Pre^^cting  local  visibility  conditions.  The  object 
V?S  t0  demonstrate  the  feasibility  of  using  this 
system  that  resulted  was  meant  to  be  only  a 
Ux  e1'  rk#  prototype.  Nevertheless,  it  was  felt  that  evalua- 

of  prototype  required  testing  under  near- 
%r>uld  Hr-  »  ^ ions,  with  comments  and  suggestions  from  users 

w-rvir«f^rVenta^ve  weather  forecasters  within  the  Air 
*t  «>*•*  letl  the  initial  prototype  was  deployed 

fn-t  r-*00*  V.°ns.  (Doveri  AFB,  Delaware  and  Seymour-Johnson 
Urw  firlrt  ^  Carolina)  shown  in  Figure  1.  The  results 

,w^ich  concentrated  on  verifying  that  the 
Iwr,  itpc'iH  accurate  forecasts  at  each  location,  have 

SW^t.  ’  ~  tlsevhere  (Stunder  et  al.,  1987a  and  Stunder  et  al.. 
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Figure  1.  Locations  for  which  fog  forecast  expert  systems  were 
developed.  Zeus  was  developed  for  Dover,  Seymour- 
Johnson,  and  Fort  Bragg.  The  system  designed  for 
Seymour- Johnson  was  adapted  to  Shaw. 

During  the  course  of  the  contract,  it  became  apparent  that 
factors  other  than  simple  verification  were  equally  important  in 
evaluating  the  feasibility  of  deploying  weather  forecast  expert 
systems  at  operational  forecast  offices  throughout  the  Air  Force. 
Additional  evaluations  were  performed  by  Geophysics  Laboratory 
personnel,  in  cooperation  with  Air  Weather  Service  forncasters  at 
Seymour-Johnson  AFB.  The  lessons  learned  thus  f\.  .wring  this 
project,  which  is  still  continuing,  have  ramificat'  t  for  other 
expert  systems  destined  for  widespread  deployment. 

2.  THE  ROLE  OF  THE  USER  IN  EXPERT  SYSTEMS 

Developers  of  expert  systems,  even  more  than  standard  computer 
programmers,  must  pay  particular  attention  to  the  user-computer 
interface.  The  umbrella  title,  "user  acceptance,"  includes 
numerous  factors  that  contribute  to  the  user's  perception  that  the 
expert  system  provides  a  useful  adjunct  to  available  computer 
products  and  displays.  Foremost  of  these,  of  course,  is  that  the 
system  produces  advice  that  the  user  finds  reasonable  and  accept¬ 
able.  The  field  program  demonstrated  that  there  are  many  factors 
entering  into  this  user  acceptance/system  accuracy  equation.  In 
hindsight,  it  might  be  argued  that  these  results  should  have  been 
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foreseen:  in  actuality,  it  is  not  possible  to  predict  user  reaction 
to  an  expert  system  without  having  the  users  evaluate  the  system 
under  conditions  closely  resembling  those  in  the  field.  Problems 
relating  to  usercomputer  interaction  fall  into  three  categories:  an 
incorrect  conceptual  model  of  the  user;  the  method  of  user  input; 
and  inappropriate  explanation  facilities. 

2.1  CONCEPTUAL  MODEL  OF  THE  USER 

The  system  was  designed  to  be  used  by  apprentice  weather  fore¬ 
casters,  some  of  whom  may  be  on  their  first  assignment  following 
forecaster  training  school.  However,  it  was  hoped  that  more 
experienced  forecasters  would  also  find  the  fog  forecast  expert 
system  of  some  help.  The  developers  of  the  system  were  research 
meteorologists;  those  interviewed  were  the  more  experienced  opera¬ 
tional  forecasters  at  each  location.  Consequently,  those  who  were 
the  most  likely  beneficiaries  of  the  system  did  not  see  it  before 
being  asked  to  exercise  it  in  the  field. 

The  first  surprise  to  the  researchers  was  the  time  it  took  to 
train  the  forecasters  in  the  use  of  the  system.  The  developers  had 
apparently  assumed  a  level  of  computer  literacy  only  slightly  lower 
than  their  own;  they  encountered  more  than  one  user  who  was  com¬ 
pletely  unfamiliar  with  personal  computers.  The  second  surprise 
was  the  realization  that  queries  asking  for  user  input  were  not 
crystal  clear,  and  the  users  asked  for  clarification.  Finally, 
some  of  the  user  input  required  a  level  of  sophistication  in 
weather  forecasting  almost  egual  to  that  of  making  the  fog  forecast 
itself. 

2.2  METHOD  OF  USER  INPUT 

In  the  initial  prototype,  all  data  were  entered  manually,  and 
queries  were  phrased  in  a  multiple-choice  format.  This  caused 
immediate  rebellion  by  the  users.  All  the  user  comments  the  first 
week  of  testing  included  objections  to  the  length  of  time  it  took 
to  enter  the  observational  data  required  to  initialize  the  expert 
system,  and  the  multiple-choice  format  of  the  non-data  entries. 

This  causes  a  bit  of  a  dilemma.  On  the  one  hand,  it  is  certainly 
not  desirable  to  remove  the  user  completely  from  the  loop,  for 
various  reasons  cited  in  the  literature  (Gordon,  1988).  However, 
weather  forecasting  is  a  time-critical  activity,  and  it  is  simply 
not  feasible  to  have  the  forecaster  chained  to  a  system  that 
constantly  demands  input.  An  appropriate  compromise  would  be  to 
have  all  numerical  data  (temperatures,  pressures,  wind  speeds  and 
directions,  etc.)  entered  automatically,  and  other  information,  por 
such  as  the  present  and  future  location  of  synoptic  weather  t 
features,  through  a  mouse  or  tablet.  Further  modifications,  such 
as  those  suggested  by  Kemp  et  al.,  (1988)  will  also  be  tested. 


2.3  EXPLANATION  FACILITIES 

All  expert  system  shells  advertise  that  they  have  an  explana¬ 
tion  facility.  What  this  often  consists  of,  as  it  did  in  this 
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instance,  is  a  listing  of  the  rules  used  by  the  system  in  reaching 
the  given  conclusion,  or  the  rule  for  which  the  system  is  asking 
information.  This  often  proved  to  be  more  exasperating  than 
informative  to  users  of  the  fog  forecast  system.  Future  plans  call 
for  a  multi-layered  explanation  capability  in  the  expert  system, 
ranging  from  presentation  of  the  rule  being  investigated,  through  a 
plain  language  statement  of  the  line  of  reasoning  being  pursued,  to 
a  tutorial  in  the  basic  physics  and  meteorology  underlying  the 
system.  Depending  on  the  time  available,  and  the  user's  inclina¬ 
tion,  these  successive  levels  of  explanation  could  be  obtained  by 
successive  "Why?"  queries,  an  explanation  capability  of  this  sort 
lies  in  the  future,  but  it  will  probably  be  an  essential  part  of 
the  installation  program  we  envision  accompanying  generic  forecast 
expert  systems  to  be  deployed  at  Air  Force  weather  stations  around 
the  world.  In  that  case,  once  the  system  is  installed,  the  instal¬ 
lation  program  could  serve  as  the  tutorial  program. 

3.  SYSTEM  ADAPTABILITY 

Even  while  the  performances  of  the  fog  forecast  prototypes  were 
being  evaluated,  doubts  were  raised  as  to  the  feasibility  of 
meteorological  expert  systems  in  a  world-wide  network.  Obviously, 
some  tailoring  must  be  required  to  install  the  system  at  each 
location;  however,  if  such  tailoring  requires  extensive  additional 
knowledge  engineering,  the  whole  concept  of  distributed  expert 
systems  becomes  impractical.  Over  what  geographic  area  is  the 
prototype  knowledge  base  valid,  requiring  only  minor  modification 
from  site  to  site?  How  many  such  systems  would  have  to  be  devel¬ 
oped  to  cover  all  locations  of  potential  interest  to  the  Air  Force? 
To  evaluate  this  aspect  of  expert  systems,  we  introduced  the 
concept  of  adaptability,  defined  as  the  ease  with  which  a  system 
designed  for  a  specific  location  can  be  modified  for  satisfactory 
performance  at  other  locations. 

3.1  MOVING  FROM  SEYMOUR-JOHNSON  TO  SHAW 

As  an  initial  test  of  its  adaptability,  the  system  designed  for 
Seymour- Johnson  Air  Force  Base  was  modified  and  evaluated  as  a 
predictor  of  fog  at  Shaw  Air  Force  Base,  South  Carolina  (see  Figure 
1).  The  modification  was  done  without  interviewing  any  of  the 
forecasters  stationed  at  Shaw,  and  the  resulting  system  showed  no 
significant  difference  in  performance  when  compared  with  the 
Seymour-Johnson  system.  These  results  have  been  reported  elsewhere 
(Dyer,  1988),  and  will  not  be  discussed  in  detail  here.  The  pro¬ 
cedure  used  may  be  of  some  interest  to  those  seeking  to  extend  the 
geographic  area  over  which  their  system  is  valid. 

The  first  step  was  a  thorough  examination  of  the  knowledge 
base.  The  version  of  the  system  at  our  disposal  consisted  of  207 
IF-THEN  rules,  which  can  be  grouped  into  the  classifications  shown 
in  Table  1. 


TABLE  1.  ANALYSIS  OF  KNOWLEDGE  BASE  FOR  FOG  FORECAST  SYSTEM 


No  Modification  Needed: 

10  common  sense 

(5%) 

21  programming  flags 

(10%) 

32  basic  meteorology 

(15%) 

(radiation  fog) 

Modification  of  Data  Interface 

Sufficient: 

74  weather  descriptors 

(35%) 

(synoptic  map) 

Modification  Needed  -  Cast  into  Template  F^rm: 

70  local  rules  of  thumb 

(35%) 

(advection  fog) 

In  all  liklihood,  most  expert  systems  dealing  with  a  geograph¬ 
ically  variable  knowledge  base  will  have  rules  falling  into  each  of 
the  categories  listed  here:  the  exact  proportions  will  vary 

according  to  programming  techniques  and  the  subject  of  the  expert 
system.  In  the  present  instance,  only  the  70  rules  comprising  the 
advective  fog  module  of  the  system  were  expressed  as  local  rules  of 
thumb.  These  were  transferred  successfully  to  Shaw  by  first  cast¬ 
ing  the  rules  of  thumb  into  template  form,  then  particularizing 
them  to  Shaw. 

An  example  should  serve  to  illustrate  the  template  method. 
Fourteen  rules  dealt  with  surface  wind  directions  favorable  (or 
unfavorable)  to  the  formation  of  advective  fog.  They  were  all  of 
the  form: 

IF  the  month  is  BLANK, 

AND  the  surface  wind  direction  is 

greater  than  XXX  degrees, 

AND  the  surface  wind  direction  is 

(RULE  A)  less  than  YYY  degrees, 

THEN  the  surface  wind  direction  is  (is  not) 
favorable  to  the  formation  of 
advective  fog. 

The  critical  wind  directions  were  determined  by  consulting 
topographic  maps,  climatological  data,  and  maps  showing  the  loca¬ 
tion  of  meteorological  stations.  Rules  that  call  for  data  from 
specific  stations  (for  example,  those  upwind  of  Seymour-Johnson) 
were  readily  modified  by  changing  the  station  designations  to  those 
more  appropriate  to  Shaw  (for  example,  those  stations  upwind  of 
Shaw).  There  was  no  deterioration  in  the  performance  of  expert 
system  when  it  was  used  to  for  cast  the  occurrence  of  fog  at  Shaw 
AFB,  rather  than  at  Seymour-Johnson  AFB.  No  attempt  has  been  made 
to  adapt  the  prototype  as  a  regional  system,  but  the  experience  of 
adapting  it  from  Seymour-Johnson  to  Shaw  indicates  that  there  is 
nothing  inherent  in  either  the  knowledge  base  or  the  architecture 
to  prevent  its  adaptation  to  many  locations  along  the  eastern  sea¬ 
board,  However,  it  is  also  apparent  that  the  template  method  of 


designing  regional  expert  systems  cannot  feasibly  be  expanded  to  a 
global  system.  It  was  decided  that  future  weather  forecast  expert 
systems  developed  at  the  Air  Force  Geophysics  Laboratory  will  be 
generic  systems,  capable  of  being  installed  by  personnel  at  each 
site. 

3.2  THE  STRUCTURE  OF  GENERIC  FORECAST  SYSTEMS 

The  proposed  generic  systems  will  have  the  structure  shown  in 
Figure  2.  Knowledge  acquisition  will  delve  more  deeply  into  the 
knowledge  base  to  determine  what  physical  and  meteorological 
principles  underlie  the  rules  of  thumb.  These  will  be  common  to 
all  systems  dealing  with  the  same  forecast  problem,  regardless  of 
the  location  for  which  the  forecast  is  to  be  made.  For  example, 
the  advective  fog  module  might  consist  of  the  following  two  rules: 


IF 

a  warm  moist  air  mass  is  advecting 
towards  the  station, 

AND 

this  air  mass  will  cool  to  its  dewpoint 

(RULE  B) 

when  it  reaches  the  station, 

AND 

there  are  no  countervening  factors 
present, 

and 

THEN 

advective  fog  will  form  over  the  station. 

IF 

advective  fog  has  formed  over  the 
station, 

(RULE  C) 

AND 

there  are  no  fog  dissipating  factors 
present  or  predicted 

THEN 

low  visibilities  will  persist. 

Rules  of  this  type  would  constitute  the  lowest  level  of  the 
structure  shown  in  Figure  2,  and  could  be  obtained  directly  from 
the  scientific  literature.  This  level  would  also  include  common- 
sense  notions  such  as  the  fact  that  strong  winds  tend  to  dissipate 
fog,  and  that  if  it  is  night,  the  sun  is  not  a  factor  in  warming 
the  air  to  a  temperature  above  its  dewpoint. 

The  second  block  in  Figure  2  represents  that  portion  of  the 
knowledge  base  considering  the  effects  of  topography  and  of  clima¬ 
tology.  For  example,  this  portion  of  a  fog  forecast  system  would 
contain  the  information  that  downslope  winds  are  countervening 
factors  in  fog  formation,  and  that  during  certain  months,  nearby 
large  bodies  of  water  are  potential  sources  of  warm  moist  air 
masses.  This  block  would  also  contain  regional  climatology: 
statements  to  the  effect  that  strong  flow  from  the  Atlantic  Ocean 
is  often  a  precursor  of  fog  along  the  east  coast  of  the  United 
States  and  that  inversion  fogs  occur  along  subtropical  west  coasts 
of  Africa  and  North  and  South  America. 

These  two  lower  blocks  would  comprise  the  expert  system 
delivered  to  the  individual  forecast  offices.  The  information 
contained  in  them  can  be  obtained  from  textbooks  and  data  bases. 
The  next  two  blocks,  containing  local  effects  and  individual  rules 
of  thumb,  would  have  to  be  tailored  for  each  location.  Present 
plans  call  for  the  development  of  an  installation  program,  with  the 
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local  forecaster  entering  in  the  latitude  and  longitude  of  the 
station,  and  the  pertinent  topographical  information  entered  from  a 
GIS  data  base.  This  could  be  supplemented  by  whatever  climatologi¬ 
cal  records  are  available  for  the  site.  Like  the  two  lower  blocks, 
this  portion  of  the  program  (labeled  "Local  Climatology  and  Topog¬ 
raphy"  in  Figure  2)  will  remain  relatively  unchanged. 

The  top  block  of  Figure  2  represents  the  local  rules  of  thumb 
and  standard  procedures  in  effect  at  a  particular  site.  The  rules 
in  this  segment  would  be  installed  during  interactive  sessions  with 
the  local  expert  forecaster.  Ideally,  this  can  be  done  by  having 
the  expert  answer  computer  queries  similar  to  those  that  would  be 
asked  by  a  knowledge  engineer.  These  questions  would  be  framed  in 
the  general  form  "How  do  you  determine  whether  or  not  the  state¬ 
ment  [an  IF  clause  in  a  given  rule]  is  correct?."  Admittedly,  this 
merely  transfers  the  knowledge  acquisition  effort  from  a  single 
team  of  knowledge  engineers  to  the  individual  weather  forecasters. 
This  may  not  be  a  reasonable  approach,  in  view  of  the  lack  of  com¬ 
puter  sophistication  exhibited  by  the  users  testing  the  prototype. 
However,  a  properly  designed  installation  program  should  allow  the 
forecaster  to  tailor  the  expert  system  for  a  particular  location. 
This  would  make  feasible  the  deployment  of  versions  of  the  system 
to  multiple  locations. 


STRUCTURE  OF  A  METEOROLOGICAL 
KNOWLEDGE  BASE 


ADDED  BY 
LOCAL  EXPERT 

FORECASTER 

INDIVIDUAL  STATION 

PRACTICE  and  INDIVIDUAL  EXPERIENCE 

RULES  OF  THUMB 

LOCAL  CLIMATOLOGY  LOCAL  RECORDS  AND 

AND  TOPOGRAPHY  WRITTEN  ADVISORIES 

EFFECTS  OF  REGIONAL 
CLIMATOLOGY  AND  TOPOGRAPHY 

METEOROLOGICAL 

DATA 

PROVIDED  BY 

EXPERT  SYSTEM 

BASIC  PHYSICAL  AND 
METEOROLOGICAL  PRINCIPLES 

SCIENTIFIC 

LITERATURE 

Figure  2.  The  structure  of  a  knowledge  base  for  generic  expert 
systems  in  meteorology.  The  lowest  layer  contains 
general  physical  and  meteorological  principles  taught  in 
the  classroom.  Progressing  upward,  each  level  of  know¬ 
ledge  becomes  more  particularized,  culminating  in  the 
rules  of  thumb  used  by  an  individual  expert  at  a  given 
location.  Typical  sources  of  the  knowledge  contained  in 
each  level  are  shown  at  the  right.  The  knowledge  con¬ 
tained  in  the  two  lower  blocks  will  constitute  the 
generic  system.  The  knowledge  of  the  two  upper  blocks 
will  be  added  by  a  local  forecaster  at  each  site. 


An  additional  benefit  of  the  architecture  shown  in  Figure  2  is 
ease  of  maintainance.  As  new  data  collection  and  analysis  facili¬ 
ties  come  on-line,  they  can  be  incorporated  into  the  system  through 


modifications  to  thi  top  block,  without  affecting  the  rules  con¬ 
tained  in  the  rest  of  the  program.  '  Similarly,  as  new  discoveries 
are  made  in  meteorology,  the  rules  of  thumb  contained  in  the  upper 
block  will  change,  also  without  affecting  the  remainder  of  the 
system.  Thus,  adaptability  will  be  achieved  over  time,  as  well  as 
location. 

4.  CONCLUSIONS 

Field  tests  of  the  fog  forecast  expert  system  has  lead  to  two 
main  conclusions.  First,  much  more  attention  must  be  paid  to  the 
user  interface.  Graphics,  mousing  capability,  menus,  light  pens, 
and  explanation  facilities  should  all  be  utilized  to  make  future 
expert  systems  as  user-friendly  as  possible.  This  is  not,  properly 
speaking,  in  the  realm  of  artificial  intelligence.  Nevertheless, 
the  success  of  any  expert  system  under  battlefield  conditions  may 
well  depend  on  how  quickly  someone  unfamilar  with  the  system  and 
with  only  minimum  computer  literacy  can  be  taught  to  use  it. 

The  second  lesson  learned  from  these  tests  is  that  expert 
systems  written  for  geographically  varying  domains  such  as  weather 
forecasting  must  be  adaptable  to  multiple  locations.  Expert  system 
use  has  progressed  beyond  the  proof-of-concept  stage,  where  an 
expert  system  that  does  a  task  well  is  considered  a  success.  Now 
we  are  entering  a  second  stage,  where  hard  looks  are  taken  at  the 
time  and  effort  invested  in  knowledge  acquisition  and  computer 
program  development.  If  the  resultant  systems  are  applicable  to 
only  a  single  location,  or  a  single  narrow  task,  they  may  not  be 
worth  the  development  effort. 
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